E2E: cleanup utility by phyllis-sy-wu · Pull Request #7171 · Shopify/cli

phyllis-sy-wu · 2026-04-02T21:33:42Z

WHY are these changes introduced?

E2E tests create apps on the Dev Dashboard that can accumulate over time — especially when tests fail mid-run, CI times out, or E2E_SKIP_CLEANUP=1 is used during development. This PR adds:

Standalone cleanup utility (scripts/cleanup.ts) — bulk cleanup of leftover apps via browser automation
E2E_SKIP_CLEANUP toggle — env var to disable per-test teardown for debugging
Separation of concerns — teardownApp (per-test, in app.ts) and cleanup.ts (bulk, standalone) have independent implementations tailored to their use cases

Naming convention: The E2E infra distinguishes two cleanup mechanisms:

teardownApp (in setup/app.ts) — per-test, automatic, runs in finally blocks, cleans up ONE app. Knows the app name and store FQDN — uses direct URLs, no discovery needed.
cleanup.ts (in scripts/) — standalone script, manual, bulk operation. Discovers ALL apps matching a pattern via dashboard pagination, discovers store installs via store switcher.

WHAT is this pull request doing?

New: `scripts/cleanup.ts`

Standalone cleanup script that finds E2E test apps on the Dev Dashboard, uninstalls them from all stores, and deletes them. Browser helpers (findAppsOnDashboard, uninstallApp, deleteApp) are self-contained in this file — not shared with app.ts.

# Shorthand (from repo root):
pnpm test:e2e-cleanup              # Full cleanup: uninstall + delete
pnpm test:e2e-cleanup -- --list    # List matching apps without action
pnpm test:e2e-cleanup -- --delete  # Delete only (skip apps still installed)

# Or directly:
npx tsx packages/e2e/scripts/cleanup.ts              # Full cleanup: uninstall + delete
npx tsx packages/e2e/scripts/cleanup.ts --list        # List matching apps without action
npx tsx packages/e2e/scripts/cleanup.ts --uninstall   # Uninstall from all stores only (no delete)
npx tsx packages/e2e/scripts/cleanup.ts --delete      # Delete only (skip apps still installed on stores)
npx tsx packages/e2e/scripts/cleanup.ts --headed      # Show browser window
npx tsx packages/e2e/scripts/cleanup.ts --pattern X   # Match apps containing "X" (default: "E2E-")

Features:

Browser-based login via shared completeLogin helper (no PTY/node-pty — works with npx tsx)
Dashboard pagination — finds apps across all pages
Install count from dashboard card text — skips installs page visit when not needed
Per-app retry with configurable max retries (default: 2)
30s action timeout (vs 60s in tests) — faster failure detection + retry
Progress UI with install counts, indented status lines, and summary (succeeded/skipped/failed)
Exports cleanupAllApps() for use as a Playwright globalTeardown in the future

UI output:

[cleanup] Mode:    Uninstall + Delete
[cleanup] Org:     161686155
[cleanup] Pattern: "E2E-"

[cleanup] Logging in...
[cleanup] Logged in successfully.
[cleanup] Navigating to dashboard...
[cleanup] Found 3 app(s)

  1. E2E-deploy-1775107268211 (0 installs)
  2. E2E-dev-1775107310343 (1 install)
  3. E2E-scaffold-1775107373607 (1 install)

[cleanup] [1/3] E2E-deploy-1775107268211
  Not installed
  Deleting...
  Deleted

[cleanup] [2/3] E2E-dev-1775107310343
  Uninstalling...
  Uninstalled
  Deleting...
  Deleted

[cleanup] [3/3] E2E-scaffold-1775107373607
  Uninstalling...
  Attempt 1 failed: Uninstall incomplete — some stores may remain
  Retry 1/2...
  Uninstalling...
  Uninstalled
  Deleting...
  Deleted

[cleanup] Complete: 3 succeeded

Updated: `setup/app.ts` — `teardownApp` rewrite

teardownApp now has its own independent implementation (no shared browser helpers with cleanup.ts):

Login — authenticates the browser via completeLogin (browser isn't authenticated on dev.shopify.com from the test's OAuth flow)
Find app — navigates to dashboard with ?search={appName} (fast, no pagination needed)
Check installs — reads install count from dashboard card text (zero extra navigation)
Uninstall (if installed) — goes directly to admin.shopify.com/store/{slug}/settings/apps via known FQDN
Delete — goes to app settings page on dev dashboard

`E2E_SKIP_CLEANUP` toggle

All test finally blocks now check this env var. When set, both local file cleanup and remote app teardown are skipped:

# Skip all cleanup — inspect apps on Dev Dashboard + local files after test run
E2E_SKIP_CLEANUP=1 DEBUG=1 pnpm --filter e2e exec playwright test

How to test your changes?

Test the cleanup script:

Create some test apps: DEBUG=1 E2E_SKIP_CLEANUP=1 pnpm --filter e2e exec playwright test app-deploy
List them: pnpm test:e2e-cleanup -- --list
Full cleanup: pnpm test:e2e-cleanup -- --headed
Verify apps are removed from dashboard

Test E2E_SKIP_CLEANUP:

Run with skip: E2E_SKIP_CLEANUP=1 DEBUG=1 pnpm --filter e2e exec playwright test app-deploy
Verify app still exists on Dev Dashboard after test completes

Test per-test teardown:

Run: DEBUG=1 pnpm --filter e2e exec playwright test app-deploy
Check DEBUG output for teardown steps (login, find, uninstall/skip, delete)
Verify app is removed from dashboard

Test error/retry handling:

To simulate failures and test the retry logic, temporarily add this in cleanup.ts before // Step 4:

// Simulate 500 errors — REMOVE AFTER TESTING
await page.route('**/apps/**/settings', (route) => {
  route.fulfill({status: 500, body: 'Internal Server Error'})
})

Examples

--list (with install counts)

[cleanup] Mode:    List only
[cleanup] Org:     161686155
[cleanup] Pattern: "E2E-"

[cleanup] Logging in...
[cleanup] Logged in successfully.
[cleanup] Navigating to dashboard...
[cleanup] Found 7 app(s)

  1. E2E-basic2-1775528925887 (0 installs)
  2. E2E-basic-1775528925887 (1 install)
  3. E2E-deploy-1775515675842 (0 installs)
  4. E2E-dev-1775515690835 (1 install)
  5. E2E-scaffold-1775515707622 (0 installs)
  6. E2E-hot-reload-1775515737862 (1 install)
  7. E2E-111 (0 installs)

--uninstall (with skips)

[cleanup] Mode:    Uninstall only
...

[cleanup] [1/3] E2E-basic2-1775528925887
  Not installed (skipped)

[cleanup] [2/3] E2E-basic-1775528925887
  Uninstalling...
  Uninstalled

[cleanup] [3/3] E2E-111
  Not installed (skipped)

[cleanup] Complete: 1 succeeded, 2 skipped

--delete (with skips and failures)

[cleanup] Mode:    Delete only
...

[cleanup] [1/7] E2E-basic-1775528925887
  Delete skipped (still installed)

[cleanup] [2/7] E2E-deploy-1775515675842
  Deleting...
  Deleted

[cleanup] [3/7] E2E-dev-1775515690835
  Delete skipped (still installed)

...

[cleanup] Complete: 2 succeeded, 3 skipped, 2 failed

Full cleanup

[cleanup] Mode:    Uninstall + Delete
...

[cleanup] [1/3] E2E-deploy-1775107268211
  Not installed
  Deleting...
  Deleted

[cleanup] [2/3] E2E-dev-1775107310343
  Uninstalling...
  Uninstalled
  Deleting...
  Deleted

[cleanup] [3/3] E2E-scaffold-1775107373607
  Not installed
  Deleting...
  Deleted

[cleanup] Complete: 3 succeeded

E2E_SKIP_CLEANUP=1 (apps remain on dashboard after test)

$ E2E_SKIP_CLEANUP=1 DEBUG=1 pnpm --filter e2e exec playwright test multi

  ✓  1 tests/multi-config-dev.spec.ts:14:3 › dev with -c flag loads the named config (19.3s)
  ✓  2 tests/multi-config-dev.spec.ts:90:3 › dev without -c flag uses default config (15.9s)
  2 passed (51.6s)

# Apps E2E-multi-cfg-* and E2E-mcfg-def-* remain on Dev Dashboard for inspection
# Clean up later with: pnpm test:e2e-cleanup

Measuring impact

n/a - this doesn't need measurement, e.g. a linting rule or a bug-fix

Checklist

I've considered possible cross-platform impacts (Mac, Linux, Windows)
I've considered possible documentation changes

phyllis-sy-wu · 2026-04-02T21:33:56Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

E2E: QA - app basic flow #7211
E2E: cleanup utility #7171 👈 (View in Graphite)
E2E: migrate tests to use new infra #7162
E2E: infra refactor + app tests update #7147
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

github-actions · 2026-04-02T22:14:28Z

Differences in type declarations

We detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:

Some seemingly private modules might be re-exported through public modules.
If the branch is behind main you might see odd diffs, rebase main into this branch.

New type declarations

We found no new type declarations in this PR

Existing type declarations

packages/cli-kit/dist/public/common/version.d.ts

@@ -1 +1 @@
-export declare const CLI_KIT_VERSION = "3.93.0";
\ No newline at end of file
+export declare const CLI_KIT_VERSION = "3.92.0";
\ No newline at end of file

This was referenced Apr 2, 2026

E2E: migrate tests to use new infra #7162

Open

E2E: infra refactor + app tests update #7147

Open

phyllis-sy-wu added the #gsd:49408 Agentic app validation label Apr 2, 2026

phyllis-sy-wu force-pushed the psyw-0402-E2E-cleanup-utility branch 2 times, most recently from 7ea8175 to d9e9bbf Compare April 2, 2026 22:12

phyllis-sy-wu force-pushed the psyw-0402-E2E-cleanup-utility branch 2 times, most recently from 5c6b425 to 7e992d2 Compare April 2, 2026 22:23

phyllis-sy-wu force-pushed the psyw-0401-E2E-migrate-remaining-tests-to-use-new-infra branch from 1d49241 to 8be6206 Compare April 2, 2026 22:24

phyllis-sy-wu force-pushed the psyw-0402-E2E-cleanup-utility branch 3 times, most recently from b83a2b2 to 42157aa Compare April 3, 2026 00:45

phyllis-sy-wu marked this pull request as ready for review April 3, 2026 00:48

phyllis-sy-wu requested a review from a team as a code owner April 3, 2026 00:48

phyllis-sy-wu force-pushed the psyw-0402-E2E-cleanup-utility branch from 42157aa to bc5f2f8 Compare April 6, 2026 19:42

phyllis-sy-wu force-pushed the psyw-0401-E2E-migrate-remaining-tests-to-use-new-infra branch from 8be6206 to 3a642b1 Compare April 6, 2026 19:42

E2E: cleanup utility + teardown refactor

14613f3

phyllis-sy-wu force-pushed the psyw-0402-E2E-cleanup-utility branch from bc5f2f8 to 14613f3 Compare April 7, 2026 03:40

phyllis-sy-wu mentioned this pull request Apr 7, 2026

E2E: QA - app basic flow #7211

Draft

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

E2E: cleanup utility#7171

E2E: cleanup utility#7171
phyllis-sy-wu wants to merge 1 commit intopsyw-0401-E2E-migrate-remaining-tests-to-use-new-infrafrom
psyw-0402-E2E-cleanup-utility

phyllis-sy-wu commented Apr 2, 2026 •

edited

Loading

Uh oh!

phyllis-sy-wu commented Apr 2, 2026 •

edited

Loading

Uh oh!

github-actions bot commented Apr 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

phyllis-sy-wu commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

WHY are these changes introduced?

WHAT is this pull request doing?

New: scripts/cleanup.ts

Updated: setup/app.ts — teardownApp rewrite

E2E_SKIP_CLEANUP toggle

How to test your changes?

Examples

Measuring impact

Checklist

Uh oh!

phyllis-sy-wu commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 2, 2026

Differences in type declarations

New type declarations

Existing type declarations

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

phyllis-sy-wu commented Apr 2, 2026 •

edited

Loading

New: `scripts/cleanup.ts`

Updated: `setup/app.ts` — `teardownApp` rewrite

`E2E_SKIP_CLEANUP` toggle

phyllis-sy-wu commented Apr 2, 2026 •

edited

Loading